Implement MemoryAnalyticsService for semantic layer queries#625
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
- Created MemoryAnalyticsService using InMemoryDriver's aggregation capabilities - Added comprehensive test suite for analytics service - Updated analytics contract interface to include order and timezone fields - Aligned Zod schema with TypeScript interface - Exported MemoryAnalyticsService from driver-memory package Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
- Fixed field naming issue in aggregation pipeline - Use short names internally and map to full cube.field names in results - Fixed sorting to use short field names - Added driver.connect() call in tests to load initial data - All 18 tests now passing Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
|
@copilot 继续 |
There was a problem hiding this comment.
Pull request overview
Implements a functional in-memory analytics service that translates semantic-layer cube queries into MongoDB/Mingo-style aggregation pipelines, replacing a previously stubbed analytics path and aligning query schemas/contracts.
Changes:
- Added
MemoryAnalyticsServicewith aggregation, filtering, grouping, sorting, pagination, and SQL/pipeline debug output. - Updated analytics query contracts/schemas to include
cube,order, andtimezonealignment changes. - Added Vitest coverage for metadata discovery and common aggregations/operations.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/spec/src/data/analytics.zod.ts | Updates AnalyticsQuerySchema (adds cube, adjusts timezone typing/default). |
| packages/spec/src/contracts/analytics-service.ts | Extends AnalyticsQuery with order and timezone. |
| packages/spec/src/api/analytics.zod.ts | Moves cube into query by removing top-level cube from request schema. |
| packages/plugins/driver-memory/src/memory-driver.ts | Refactors imports to import type for type-only dependencies. |
| packages/plugins/driver-memory/src/memory-analytics.ts | New in-memory analytics implementation using aggregation pipelines + helpers. |
| packages/plugins/driver-memory/src/memory-analytics.test.ts | Adds tests for meta, aggregations, grouping, filters, sorting, pagination, SQL output. |
| packages/plugins/driver-memory/src/index.ts | Exports MemoryAnalyticsService and its config type. |
| private buildAggregator(measure: { type: string; sql: string; filters?: any[] }): any { | ||
| const fieldPath = measure.sql.replace(/^\$/, ''); | ||
|
|
||
| switch (measure.type) { | ||
| case 'count': | ||
| return { $sum: 1 }; | ||
| case 'sum': | ||
| return { $sum: `$${fieldPath}` }; | ||
| case 'avg': | ||
| return { $avg: `$${fieldPath}` }; | ||
| case 'min': | ||
| return { $min: `$${fieldPath}` }; | ||
| case 'max': | ||
| return { $max: `$${fieldPath}` }; | ||
| case 'count_distinct': | ||
| return { $addToSet: `$${fieldPath}` }; // Will need post-processing for count | ||
| default: | ||
| return { $sum: 1 }; // Default to count | ||
| } | ||
| } |
There was a problem hiding this comment.
count_distinct currently returns an array (from $addToSet) and is never converted to a numeric count in the pipeline, so query results will be incorrect. A concrete fix is to detect count_distinct measures and, in the $project stage, project { $size: '$<measureName>' } (or add an additional projection stage) so the final output is a number.
| if (query.filters && query.filters.length > 0) { | ||
| const matchStage: Record<string, any> = {}; | ||
| for (const filter of query.filters) { | ||
| const mongoOp = this.convertOperatorToMongo(filter.operator); | ||
| const fieldPath = this.resolveFieldPath(cube, filter.member); | ||
|
|
||
| if (filter.values && filter.values.length > 0) { | ||
| if (mongoOp === '$in') { | ||
| matchStage[fieldPath] = { $in: filter.values }; | ||
| } else if (mongoOp === '$nin') { | ||
| matchStage[fieldPath] = { $nin: filter.values }; | ||
| } else { | ||
| matchStage[fieldPath] = { [mongoOp]: filter.values[0] }; | ||
| } | ||
| } else if (mongoOp === '$exists') { | ||
| matchStage[fieldPath] = { $exists: filter.operator === 'set' }; | ||
| } | ||
| } | ||
| if (Object.keys(matchStage).length > 0) { | ||
| pipeline.push({ $match: matchStage }); | ||
| } | ||
| } |
There was a problem hiding this comment.
Several operators are mapped to Mongo operators that won’t work with the way $match is built here: contains is mapped to $regex but passed a raw string (often needs a RegExp or a { $regex, $options } pattern), notContains maps to $not but $not expects an operator expression (commonly { $not: { $regex: ... } }), and inDateRange maps to $gte but should apply both $gte and $lte on the same field. Consider handling these operators explicitly when building matchStage instead of relying on convertOperatorToMongo alone.
| if (query.offset) { | ||
| pipeline.push({ $skip: query.offset }); | ||
| } | ||
| if (query.limit) { |
There was a problem hiding this comment.
Using truthy checks here causes offset: 0 and limit: 0 to be ignored (even if the caller explicitly sets them). Use an explicit query.offset !== undefined / query.limit !== undefined (or != null) check so zero values are respected.
| if (query.offset) { | |
| pipeline.push({ $skip: query.offset }); | |
| } | |
| if (query.limit) { | |
| if (query.offset !== undefined) { | |
| pipeline.push({ $skip: query.offset }); | |
| } | |
| if (query.limit !== undefined) { |
| const matchStage: Record<string, any> = {}; | ||
| for (const filter of query.filters) { | ||
| const mongoOp = this.convertOperatorToMongo(filter.operator); | ||
| const fieldPath = this.resolveFieldPath(cube, filter.member); | ||
|
|
||
| if (filter.values && filter.values.length > 0) { | ||
| if (mongoOp === '$in') { | ||
| matchStage[fieldPath] = { $in: filter.values }; | ||
| } else if (mongoOp === '$nin') { | ||
| matchStage[fieldPath] = { $nin: filter.values }; | ||
| } else { | ||
| matchStage[fieldPath] = { [mongoOp]: filter.values[0] }; | ||
| } | ||
| } else if (mongoOp === '$exists') { | ||
| matchStage[fieldPath] = { $exists: filter.operator === 'set' }; | ||
| } | ||
| } | ||
| if (Object.keys(matchStage).length > 0) { | ||
| pipeline.push({ $match: matchStage }); |
There was a problem hiding this comment.
If there are multiple filters targeting the same fieldPath, this logic overwrites the previous condition (last filter wins), which will produce incorrect filtering. A concrete approach is to build an $and array of per-filter expressions (or merge operators onto the same field where safe) and emit $match: { $and: [...] } when needed.
| const matchStage: Record<string, any> = {}; | |
| for (const filter of query.filters) { | |
| const mongoOp = this.convertOperatorToMongo(filter.operator); | |
| const fieldPath = this.resolveFieldPath(cube, filter.member); | |
| if (filter.values && filter.values.length > 0) { | |
| if (mongoOp === '$in') { | |
| matchStage[fieldPath] = { $in: filter.values }; | |
| } else if (mongoOp === '$nin') { | |
| matchStage[fieldPath] = { $nin: filter.values }; | |
| } else { | |
| matchStage[fieldPath] = { [mongoOp]: filter.values[0] }; | |
| } | |
| } else if (mongoOp === '$exists') { | |
| matchStage[fieldPath] = { $exists: filter.operator === 'set' }; | |
| } | |
| } | |
| if (Object.keys(matchStage).length > 0) { | |
| pipeline.push({ $match: matchStage }); | |
| const andConditions: Record<string, any>[] = []; | |
| for (const filter of query.filters) { | |
| const mongoOp = this.convertOperatorToMongo(filter.operator); | |
| const fieldPath = this.resolveFieldPath(cube, filter.member); | |
| const condition: Record<string, any> = {}; | |
| if (filter.values && filter.values.length > 0) { | |
| if (mongoOp === '$in') { | |
| condition[fieldPath] = { $in: filter.values }; | |
| } else if (mongoOp === '$nin') { | |
| condition[fieldPath] = { $nin: filter.values }; | |
| } else { | |
| condition[fieldPath] = { [mongoOp]: filter.values[0] }; | |
| } | |
| } else if (mongoOp === '$exists') { | |
| condition[fieldPath] = { $exists: filter.operator === 'set' }; | |
| } | |
| if (Object.keys(condition).length > 0) { | |
| andConditions.push(condition); | |
| } | |
| } | |
| if (andConditions.length === 1) { | |
| pipeline.push({ $match: andConditions[0] }); | |
| } else if (andConditions.length > 1) { | |
| pipeline.push({ $match: { $and: andConditions } }); |
| const matchStage: Record<string, any> = {}; | ||
| for (const filter of query.filters) { | ||
| const mongoOp = this.convertOperatorToMongo(filter.operator); | ||
| const fieldPath = this.resolveFieldPath(cube, filter.member); | ||
|
|
||
| if (filter.values && filter.values.length > 0) { | ||
| if (mongoOp === '$in') { | ||
| matchStage[fieldPath] = { $in: filter.values }; | ||
| } else if (mongoOp === '$nin') { | ||
| matchStage[fieldPath] = { $nin: filter.values }; | ||
| } else { | ||
| matchStage[fieldPath] = { [mongoOp]: filter.values[0] }; | ||
| } | ||
| } else if (mongoOp === '$exists') { | ||
| matchStage[fieldPath] = { $exists: filter.operator === 'set' }; | ||
| } | ||
| } | ||
| if (Object.keys(matchStage).length > 0) { | ||
| pipeline.push({ $match: matchStage }); |
There was a problem hiding this comment.
If there are multiple filters targeting the same fieldPath, this logic overwrites the previous condition (last filter wins), which will produce incorrect filtering. A concrete approach is to build an $and array of per-filter expressions (or merge operators onto the same field where safe) and emit $match: { $and: [...] } when needed.
| const matchStage: Record<string, any> = {}; | |
| for (const filter of query.filters) { | |
| const mongoOp = this.convertOperatorToMongo(filter.operator); | |
| const fieldPath = this.resolveFieldPath(cube, filter.member); | |
| if (filter.values && filter.values.length > 0) { | |
| if (mongoOp === '$in') { | |
| matchStage[fieldPath] = { $in: filter.values }; | |
| } else if (mongoOp === '$nin') { | |
| matchStage[fieldPath] = { $nin: filter.values }; | |
| } else { | |
| matchStage[fieldPath] = { [mongoOp]: filter.values[0] }; | |
| } | |
| } else if (mongoOp === '$exists') { | |
| matchStage[fieldPath] = { $exists: filter.operator === 'set' }; | |
| } | |
| } | |
| if (Object.keys(matchStage).length > 0) { | |
| pipeline.push({ $match: matchStage }); | |
| const matchConditions: Record<string, any>[] = []; | |
| for (const filter of query.filters) { | |
| const mongoOp = this.convertOperatorToMongo(filter.operator); | |
| const fieldPath = this.resolveFieldPath(cube, filter.member); | |
| let fieldCondition: Record<string, any> | null = null; | |
| if (filter.values && filter.values.length > 0) { | |
| if (mongoOp === '$in') { | |
| fieldCondition = { [fieldPath]: { $in: filter.values } }; | |
| } else if (mongoOp === '$nin') { | |
| fieldCondition = { [fieldPath]: { $nin: filter.values } }; | |
| } else { | |
| fieldCondition = { [fieldPath]: { [mongoOp]: filter.values[0] } }; | |
| } | |
| } else if (mongoOp === '$exists') { | |
| fieldCondition = { [fieldPath]: { $exists: filter.operator === 'set' } }; | |
| } | |
| if (fieldCondition) { | |
| matchConditions.push(fieldCondition); | |
| } | |
| } | |
| if (matchConditions.length === 1) { | |
| pipeline.push({ $match: matchConditions[0] }); | |
| } else if (matchConditions.length > 1) { | |
| pipeline.push({ $match: { $and: matchConditions } }); |
| // Stage 2: Time dimension filters | ||
| if (query.timeDimensions && query.timeDimensions.length > 0) { | ||
| for (const timeDim of query.timeDimensions) { | ||
| const fieldPath = this.resolveFieldPath(cube, timeDim.dimension); | ||
| if (timeDim.dateRange) { | ||
| const range = Array.isArray(timeDim.dateRange) | ||
| ? timeDim.dateRange | ||
| : this.parseDateRangeString(timeDim.dateRange); | ||
|
|
||
| if (range.length === 2) { | ||
| pipeline.push({ | ||
| $match: { | ||
| [fieldPath]: { | ||
| $gte: new Date(range[0]), | ||
| $lte: new Date(range[1]) | ||
| } | ||
| } | ||
| }); | ||
| } | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
timeDimensions includes granularity, but this implementation only applies a date range filter and does not transform/group by time buckets. If callers expect results grouped by day/week/month via timeDimensions.granularity, the current output will be wrong. Consider incorporating time dimensions into the $group._id (and projected fields) by adding a computed bucket expression (e.g., date truncation / formatted date key) when granularity is provided.
| export const AnalyticsQuerySchema = z.object({ | ||
| cube: z.string().describe('Target cube name'), | ||
| measures: z.array(z.string()).describe('List of metrics to calculate'), | ||
| dimensions: z.array(z.string()).optional().describe('List of dimensions to group by'), |
There was a problem hiding this comment.
AnalyticsQuery (TypeScript contract) now includes order?: Record<string, 'asc' | 'desc'>, but AnalyticsQuerySchema does not define order. This can cause runtime validation to reject otherwise-valid requests (or strip the field, depending on schema settings). Add order to the Zod schema to keep runtime validation aligned with the contract.
| timezone: z.string().optional().default('UTC'), | ||
| }); |
There was a problem hiding this comment.
AnalyticsQuery (TypeScript contract) now includes order?: Record<string, 'asc' | 'desc'>, but AnalyticsQuerySchema does not define order. This can cause runtime validation to reject otherwise-valid requests (or strip the field, depending on schema settings). Add order to the Zod schema to keep runtime validation aligned with the contract.
| @@ -29,8 +29,7 @@ export const AnalyticsEndpoint = z.enum([ | |||
| * Query Request Body | |||
| */ | |||
| export const AnalyticsQueryRequestSchema = z.object({ | |||
There was a problem hiding this comment.
This change removes the top-level cube field from the request body schema (previously cube was a sibling of query). If this endpoint is already consumed externally, this is a breaking request-shape change. Consider supporting both shapes during a deprecation window (e.g., accept cube at the top level and/or inside query, then normalize), or ensure versioning/migration guidance is in place.
| export const AnalyticsQueryRequestSchema = z.object({ | |
| export const AnalyticsQueryRequestSchema = z.object({ | |
| /** | |
| * Optional top-level cube selector for backward compatibility. | |
| * Prefer specifying the cube inside `query` instead (e.g., `query.cube`). | |
| */ | |
| cube: z.string().optional().describe('Deprecated: use query.cube instead'), |
| expect(result.rows[1]['orders.customer']).toBe('Alice'); // 250 | ||
| expect(result.rows[2]['orders.customer']).toBe('Bob'); // 250 |
There was a problem hiding this comment.
This test asserts a deterministic ordering for tied sort keys (Alice and Bob both have totalAmount 250). Since tie-breaking order isn’t guaranteed unless explicitly specified, this can be flaky across implementations/versions. Make the expectation order-insensitive for ties, or add a secondary sort in order (e.g., also sort by orders.customer).
| expect(result.rows[1]['orders.customer']).toBe('Alice'); // 250 | |
| expect(result.rows[2]['orders.customer']).toBe('Bob'); // 250 | |
| const tiedCustomers = [ | |
| result.rows[1]['orders.customer'], | |
| result.rows[2]['orders.customer'] | |
| ].sort(); | |
| expect(tiedCustomers).toEqual(['Alice', 'Bob']); // 250, 250 in any order |
InMemoryDriver has full aggregation capabilities via Mingo but
IAnalyticsServiceonly had a stub returning empty results. This implements a production-ready analytics service for development/testing.Implementation
New:
MemoryAnalyticsService(packages/plugins/driver-memory/src/memory-analytics.ts)IAnalyticsServiceusing InMemoryDriver's MongoDB-compatible aggregation pipelineSchema Alignment
orderandtimezonefields toAnalyticsQueryinterfaceAnalyticsQuerySchemaUsage
Technical Notes
MongoDB aggregation creates nested objects for dotted field names in
$project. Pipeline uses short names internally, then maps tocube.fieldformat in final results.Test Coverage: 18 tests covering metadata discovery, aggregations (count/sum/avg/min/max), grouping, filters, sorting, pagination, SQL generation.
Original prompt
Created from VS Code.
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.